[recent work](https://www.lesswrong.com/posts/zfgQCdnMBa3hBdzpL) on [latent introspective awareness](https://transformer-circuits.pub/2025/introspection/index.html) has put something of a damper on introspection, basically due to a combination of the work not moving far beyond qualitative data (initial experiments were freeform writing for a while!), and not effectively ruling out [causal bypassing](https://www.lesswrong.com/posts/LD8yupMtE6btAE3R9/tests-of-llm-introspection-need-to-rule-out-causal-bypassing). from this, it might make sense to conclude that introspection is not really possible nor a desirable thing for llms

at least from current evidence I am willing to believe introspection isn't really possible. i think a long-standing confusion I had towards introspection is that [bitter-lesson-pilled](https://www.lesswrong.com/posts/rwoEz3bA9ekxkabc7/activation-oracles-training-and-evaluating-llms-as-general) interp (trying to get natural language explanations from activations) could basically implement the same type of capabilities as introspection, and that it was easy to select for this sort of interp -- people had already done so by very low-rank fine-tuning, and steering vectors to some extent -- so introspective interp must also be easy. but now this seems obviously false, since... input space is very different from latent space. i would be hard pressed to name any examples of llm capabilities that involve, in a single forward pass, reading off some internal activation state and exactly reporting it (rather than just acting from latent states)... this excludes introspection via reading off [kv caches from prior tokens](https://xcancel.com/repligate/status/1963460961744163145), which doesn't seem explored very much, though it's not clear what non-obvious things this type of introspection would enable, and this type seems more straightforward to explain.

but introspection in general *does* seem desirable to me. at first glance it doesn't seem obvious what good things latent introspection would enable. how exactly does this enable better thinking and strategy, instead of just having models use interp on themselves effectively? one broad intuition in favor of introspection is that models probably know their world model and best ontology better than us, and having models probe their own world model for states is probably better and more scalable. nothing says much, if anything about this outcome; current bitter-lesson-pilled interp has shown pretty underwhelming results, and almost definitely worse than humans explaining with standard interp stuff. my corresponding theory of change was something like:

> llms probably know their world model really well in principle, but this knowledge is strongly underelicited. so let's find ways to elicit different types of introspection strategies (as demonstrated in latent introspective awareness) so we can elicit more understanding of the world model, and steer towards llms better reflecting on thinking/strategy/stuff that llms [seem poor at by default](https://www.lesswrong.com/posts/WewsByywWNhX9rtwi/current-ais-seem-pretty-misaligned-to-me) which would be good for safety

strong non-scheming introspection does seem desirable imv but it seems hard to get, especially with these new explanations of supposed introspective capabilities. i think this puts a dent in my theory of change, and it might be somewhat less likely to outright impossible that it works out; some alternate outcomes in order of least difficulty: (1) it still roughly [exists](https://arxiv.org/abs/2512.12411), or swapping in introspection via kv cache is actually robust, and this is ok since kv caching is used ~everywhere, (2) it is really hard for gradient descent to select models implementing strong general introspection/reflection since it's computationally expensive (roughly same reason as [inner search](https://www.lesswrong.com/posts/TdesHi8kkyokQdDoQ/gradient-descent-doesn-t-select-for-inner-search)?), but something is figured out somehow, (3) eliciting introspection does not meaningfully improve on world model understanding for llms, (4) true introspection just doesn't exist in transformers and we have to continue with trying to traverse world models ourselves, or eliciting better reasoning by other (probably less effective) means

one can argue that stuff like activation oracles are strictly better than introspection, as it already involves tuning a model towards interpreting its own activations, while factoring its own world model -- as an AO is just a fine-tune of the model being interpreted, and the tokens the AO produces is based on the base model's priors. currently this is true and i think it's wrong to trust llm self-reports over even AOs, but i think the objective that could specify this capability is almost always going to be followed in a very path-of-least-resistance way, relying on a quite shallow self-understanding of the world model. this goes back to how we should best elicit self-understanding of the world model, and i think a plausible route is general introspection through a self-modelling objective. basically just let models converge onto a more general solution (despite it being more computationally expensive than learning a bag of heuristics). there might already be some ground for a weak self-modelling objective when llms observe their own writing in pre-training, associate it with some 'LLM' persona, and then select all of the info associated with the persona when asked to represent that character in [constitutional ai](https://arxiv.org/abs/2212.08073). this is perhaps analogous to how models like [Qwen3-32B can predict its persona](https://www.lesswrong.com/posts/3qEDD8Sn7XoNGeGhm/to-what-extent-is-qwen3-32b-predicting-its-persona), but this likely lacks causal mechanisms that produce latent introspective awareness, and i don't think current pressures are deliberate enough that models are learning introspective machinery or whatever

beyond involving quite expensive computational machinery, i think (3) is probably hard to specify, since introspection does not seem like a convergent thing to implement for neutral bets in general, unlike inner search processes. an equivalent problem for self-modelling seems somewhat like models converging on a particular persona after post-training (inheriting a bag of heuristics as priors for generating tokens) vs. introspective machinery that considers significantly more bits of info which might translate to more uncovered heuristics, or more [motivations](./motivations.md). or maybe more simply, models can literally read some approximation of prior internal activations (within the forward pass, or from prior fps via e.g. kv cache) which reveals more info. maybe introspection is a superset of inner search in some way

this sort of desired introspection might also decompose into a bunch of related myopia problems and maybe it's best to look at something like [recursive forecasting](https://www.lesswrong.com/posts/q2zYtNsh62SCphitt/recursive-forecasting-eliciting-long-term-forecasts-from) and other methods for eliciting non-myopia from myopic processes (and roughly the vice-versa, eliciting myopia from non-myopic processes, like [MONA](https://www.lesswrong.com/posts/zWySWKuXnhMDhgwc3/mona-managed-myopia-with-approval-feedback-2))

here is a sort of tech tree of strategies that llms may use to acquire more knowledge and update on priors, a la "introspection"

![600](./PXL_20260702_183456502~2.jpg)

i don't think it's obvious or true that "(arbitrary activation) -> summary -> feature" circuits is the result of any persona-related process, and i think it's more likely that such general structure would arrive from incentives to learn inner search-like processes, even if it's less likely in general that this strong introspection would exist in llms. i also wish this tree wasn't anchored so heavily on personas but it does seem like the best way to [reason about](https://www.lesswrong.com/posts/DTDoyDTtC8R3bCiTx/from-personas-to-intentions-towards-a-science-of-motivations) [current systems](./motivations.md)

in general ive updated towards introspection being less likely (~85% -> 65-70%)